134 Feedforward Neural Networks
generalization performance depends on the size and efficiency of the
training set, besides the architecture of the network and the
complexity of the problem [Hush and Horne, 19931. Testing the
performance of the network with new data is called cross-validation.
If the performance for the test data is as good as for the training
data, then the network is said to have generalized from the training
data. Further discussion on generalization is given later in Section
7.3 and in Appendix D.
Tasks with backpropagation network: A backpropagation network
can be used for several applications such as realization of logic
functions, pattern classification, pattern mapping, function approxi-
mation, estimation of probability distribution and prediction [Hush
and Horne, 19931. These tasks were demonstrated in several real
world applications such as in speech, character recognition, system
identification, passive sonar speech synthesis,
etc. [Sejnowski and Rosenberg, 1987;Cohen et al, 1993; et al,
1990;Narendra and Parthasarathy, 1990;Casselman et al, 19911.
Limitations of backpropagation: The major limitation of the back-
propagation learning is its slow convergence. Moreover, there is no
proof of convergence, although it seems to perform well in practice.
Due to stochastic gradient descent on a nonlinear error surface, it is
likely that most of the time the result may converge to some local
minimum on the error surface and Tesi, 19921.There is no easy
way to eliminate this effect completely, although stochastic learning
algorithms were proposed to reduce the effects of local minima
19881.Another major problem is the problem of scaling.
When the complexity of the problem is increased, there is no
guarantee that a given network would converge, and even if it
converges, there is no guarantee that good generalization would
result. The complexity of a problem can be defined in terms of its
size or its predicate order and 1990;Hush and Horne,
19931.Effects of scaling can be handled by using the prior information
of the problem, if possible. Also, modular architectures can also
reduce the effects of the scaling problem [Ballard, 1990;Jacobs et al,
1991; 19941.
For many applications, the desired output may not be known
precisely. In such a case the backpropagation learning cannot be used
directly. Other learning laws have been developed based on the
information whether the response is correct or wrong. This mode of
learning is called reinforcement learning or learning with critic
[Sutton et al, 1991; 19921 as discussed in Section 2.4.6.
Extensions of backpropagation: Principles analogous to the ones
used in the backpropagation network have been applied to extend the